Skip to content

[WIP] feat: [#1096] While, Do-while loop implementation for fetching paginated data from an API #1097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

nyamathshaik
Copy link
Contributor

@nyamathshaik nyamathshaik commented Apr 14, 2025

Please specify parts of this PR update:

What this PR does:
This PR introduces enhancements to support iterative logic within the do block using a while loop construct. Specifically, it adds support for:

A while condition to allow loop execution as long as the condition evaluates to true.

postConditionCheck: false to enable do-while loop semantics.

maxIterations to define an upper limit on loop executions, providing safety against infinite loops.

The changes are accompanied by an example paginated-fetch-example that demonstrates fetching paginated API data until .hasNextPage == true, accumulating results across iterations.

Additional information:
This capability enables richer control flow for iterative data-fetching use cases in serverless workflows, reducing the need for complex custom code and improving the expressiveness of the DSL.

Sample document:

document:
  dsl: '1.0.0'
  namespace: test
  name: paginated-fetch-example
  version: '0.1.0'

do:
  - fetchPaginatedData:
      condition: .hasNextPage == true
      isDoWhile: true # Enables do-while behavior
      limit: 100
      do:
        - fetchPage:
            call: getPageData
            input:
              pageToken: .nextPageToken
            output: .fetchedData
        - accumulateData:
            run: mergeResults
            input:
              newData: .fetchedData
              existingData: .accumulatedResults
            output: .accumulatedResults

@cdavernas
Copy link
Member

cdavernas commented Apr 14, 2025

Hi, @nyamathshaik and thanks for your work!

However, as per contribution guidelines, I have to ask you to please refrain from opening a PR, especially not in draft, before gaining approval from all maintainers, as your feature request might be rejected (even though, in this case, it probably won't be, but the implementation details still remain to be defined)

@cdavernas
Copy link
Member

I have a couple of remarks regarding your proposal, which I provided in the related issue.

@nyamathshaik
Copy link
Contributor Author

nyamathshaik commented Apr 14, 2025

@cdavernas Thanks for letting me know! 😊 Could you please take a look at the issue #1096 and let me know if the proposal looks good to move forward?

Happy to incorporate any feedback or iterate further based on the maintainers' suggestions.

@ricardozanini
Copy link
Member

@nyamathshaik I think this is a WIP, right? Since we haven't settled the conversation yet.

@nyamathshaik nyamathshaik changed the title feat: [#1096] While, Do-while loop implementation for fetching paginated data from an API [WIP] feat: [#1096] While, Do-while loop implementation for fetching paginated data from an API Apr 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants